DOCUMENT RESUME 



ED 062 386 



TM 001 320 



AUTHOR 

TITLE 

INSTITUTION 
SPONS AGENCY 



PUB DATE 

GRANT 

NOTE 



Stuff lebeam, Daniel L. 

Design of a Planning and Assessment System for the 
Division of Manpower and Institutions. 

Ohio State Univ. , Columbus. Evaluation Center.; Ohio 
State liniv. , Columbus. Research Foundation. 

National Center for Educational Research and 
Development (DHEW/OE) , Washington, D. C. Panel on 
Educational Terminology. 

18 Jun 71 
OEG-0-7 1-4558 

16p. 



EDRS PRICE 
DESCRIPTORS 

IDENTIFIERS 



MF-$0. 65 HC— $3. 29 

Design Needs; *Evaluation Methods; *Management 
Systems; Models; Program Budgeting; *Program Design; 
^Program Proposals 

♦Division of Manpower and Institutions 



ABSTRACT 

This is a proposal submitted by the Ohio State 
University Research Foundation for design of a planning and 
assessment system for the Division of Manpower and Institutions (now 
Division of Research and Development Resources), U.S. Office of 
Education. It responds to a need for an expanded planning, 
management, and assessment system that would enable DMI leaders to 
make responsible decisions under its new institutional support 
policy. The proposal suggests a background study* the convening of 
two advocate teams to develop alternative evaluation strategies; a 
convergence team to review the competing strategies and generate a 
third, which could be an elaboration of one of the advocate team 
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Background 

The United States Office of Education, through the Division of 
Manpower and institutions, currently supports 27 educational R S 0 
institutions with an annual budget of approximately $34 million. 

Because the research-development-field testing-dissemination sequence, 
around which many of the institutions build their programs, calls 
for progressively larger budgets, DM1 has found it necessary, in a 
period of level funding, to withdraw support from some of the weaker 
institutions in order to maintain a minimum level of support in other, 
stronger institutions. 

A new support policy has emerged which is designed to reverse the 
trend toward attrition and to provide for needed new institutions and 
programs. Details of this new policy are provided in the attached 
document: DM I Institutional Support Policy (Frye, June 3» 1371). 

The new pol icy proposes a maturity model of educational R 6- D 
institutions whose chief features are summarized in Figure 1. 

The new model specifies 10 clusters of decisions which must be 
made by DM I leaders. These are indicated in Figure 1 by the numbers 
enclosed in parentheses. The key questions in each decision cluster are 
as follows: 

Pre- Inst i tutional Phase 

(1) Should given requests for planning grants for new 
institutions be funded? 

(2) Should given proposals to start new institutions 
be funded? 

Phase I (new institutions) 

(3) At the end of this phase, is the baste program plan of a 
given institution satisfactory? 

Phase II (developing Institutions) 

(4) At the end of this phase, is the capability of the 
institution to manage its basic program plan 
satisfactory? 
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STAGES AND FVENTS OF INSTITUTIONAL GROWTH IN THE MATURITY MODEL 
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Phase III (mature institutions) 



(5) Should given developmental grant proposals (for new 
program plans) be funded? 

(6) Should the Office of Education commit itself to support 
given proposed new program plans? 

(7) Are significant milestones in the program plans being 
reached? on time? with satisfactory qua 1 ity? 

(8) Did the program reach its objectives? 

(Answers to this question, of course, have significance 
beyond the information needs of DMI, e.g., OPE, NCEC, 
and others. 

(9) Do current circumstances of the programs or institution 
require redirection or changes in Office of Education 
support? 

(10) With given funds each year, how should the program be 
adjusted? stretched? compressed? 



Problem and Objectives 

In order for DMI leaders to make responsible decisions in the 10 
areas listed above, a new, expanded planning, management, and assess- 
ment system is needed within DMI. A number of internal position papers 
have already specified certain characteristics of this new system. 
However, the nature of the task requires skills and disciplinary 
qualifications not now available within DMI, Accordingly, outside 
help will be required in the further development of the system. The 
present need, thus, is for the systematic engagement of a number of 
outstanding planning, management, and assessment personnel to assist 
DMI leaders: 

In completing a background study which lays out the 
characteristics, needs, and opportunities of the new 
DMI planning, management, and assessment system. 

In preparing detailed specifications for the design 
of the new system. 

4 



( 1 ) 



( 2 ) 



O 

ERIC 



5 



(3) In designing two alternative planning and assessment 
systems for DM I, complete with detailed specifications 
for all needed instrumentation. 

(4) In selecting or synthesizing the final DM I system. 

(5) In completing pilot instrumentation according to the 
specifications. 

(6) Orientation of DMI staff concerning the new system. 

The present contract calls for only the first steps on this 
overall workplan, specifically, completion of the background study, 
preparation of specifications for the new system, and recruiting and 
orientating members of the design teams. 



Task Areas. Task Groups, and Products 

Figure 2 outlines a general strategy for the development of a new 
DMI planning and assessment system. The horizontal dimension of the 
chart identifies the major task areas and task groups within the strategy. 
The vertical dimension is a time line which has been divided Into five 
major phases. It Is reiterated that this Initial proposal covers only 
the tasks and budget for Phase I, plus Step 9 which is included in 
Phase II. Moreover, this initial proposal is being submitted under the 
assumption that additional funds will be made available to The Ohio 
State University Research Foundation during August of 1971 for the 
completion of Phases II through V of the total project. The remainder 
of this section contains definitions of the task areas and task groups 
and a description of the work . j be performed by the different task 
groups during each of the project phases. 

The background study depicted in the left hand column will be 
devoted to a study of needs to be met by the new system, as wel I as 
opportunities that could be brought to bear on meeting these needs, and 
particular tasks which must be accomplished In operationalizing a new 
strategy. 

The decision-maker column denotes functions to be performed by the 
group of persons who will make decisions about the adoption and imple- 
mentation of a new system and a team of synthesizers to serve the former 
group in formulating and articulating decisions that will need to be 
made In the selection and Implementation of the new approach. This team 
of synthesizers Is conceived to include one person from the group of 
Office of Education personnel who will make decisions about the new 
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system as well as an outside person who qualifies as an educational 
statesman and a person who is highly knowledgeable of decision making 
and evaluation regarding DM I programs. It is suggested that a person 
such as Dr. Francis Chase serve in this latter capacity. 

The fourth major task area is that of generation and evaluation of 
alternative planning and assessment systems for DM I • This area is to 
be manned by three subgroups. 

The first is a study coordinator and staff. This team will 
structure and implement a study to generate and assess competing 
system alternatives for DMI. The team will include an overall 
coordinator, an information specialist to organize, store, and retrieve 
information needed in the study of alternative systems, a measurement 
and statistics specialist to assist in the data gathering and analysis 
activities, and a technical writer to assist in the writing up of 
alternative system strategies. 

The second subgroup includes two advocate teams. These teams 
will be selected because they represent competing positions with 
respect to what kind of system should be generated to serve DMI 
purposes. Tentative lists of persons for the teams are: 



Team A 

Roald F. Campbell, Chairman 
Michael Scriven 
Robert E. Stake 
Gene V. Glass 
C. M. Lindval 1 



Team B 

Henry M. Bricked, Chairman 

Egon G. Guba 

Wi 1 1 iam B. Michael 

John E. Seger 

Howard 0. Merrlman 



The third subgroup includes a convergence team. This team is to be 
composed of persons who represent DMI and evaluation and decision making 
in Labs and Centers. The purpose of this team is to review the competing 
alternative system strategies produced by each of the advocate teams 
and to generate a third strategy which may be an elaboration of one 
of the advocate team strategies, a completely new strategy, or a merging 
of the best features of the two competing strategies. A suggested list 
of members to serve on this team includes Charles Frye, David L. Clark, 
Arthur Lumsdaine, Robert Randall, and Walter Marks. 

The last major column on the chart includes an advisory group to 
provide DMI with continuing assistance during the course of this study 
with respect to their ongoing activities. Proposed members of this 
group include Daniel L. Stuff lebeam, J. Thomas Hastings, and Richard 
Jaeger. 



Project Phases 



The work of the above Identified group Is projected to occur during 
a six and one-half month period (June 28 through January 15). The 
specific tasks to be performed during the five phases are numbered 
consecutively through the master chart . 

Phase One (two months) 

The main purpose of Phase i is to determine the main needs which 
should be met by the new planning and assessment system, and to Identify 
and assess opportunities which could be used by DM I in meetin g these 
needs . 

(1) The internal and external synthesizers will work with the 
project staff and with leaders of the design teams to clarify questions 
which should be addressed in the Initial background study In order to 
identify needs and opportunities concerning development of the new 
system (Report No. 1), 

(2) The DM t decision-making group will review and revise or extend, 
if necessary, the specifications for the background study and finally 
provide a confirmation of what Is to be produced by the study. 

(3) The team to perform the background study will produce Report 
No. 2, which Is to be a completed design for conduct of the Initial 
background study. This design must respond directly to the questions, 
constraints, and specifications identified in 1 and 2 above. 

(4) At the same time that the decision makers and study team are 
clarifying questions and a design for the background study, the coordinator 
for the study to generate and assess alternative evaluation systems 

for DMI will recruit members for the advocate and convergence teams. 

(5) The synthesizers for the decision-making group will review 
the design for the Initial background study to insure that it does 
respond adequately to the decision makers* questions and specifications. 

(6) The background study team will implement their design. 

(7) The background study team will report their findings to Ihe 
synthesizers and OE decision makers within the overall decision-making 
group (Report No. 3). 

(8) At the same time that the preceding activities are ongoing, the 
special advisory panel mentioned in the right hand column of the master 
chart will work with DMI staff as needed to assist in the implementation 
of the present DMI system. 
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It is to be noted that during Phase I various participants in and 
audiences for the DM I evaluation system— Off ice of Education decision 
makers, experienced site visitors, members of the Lab and Center staffs, 
and others— will be surveyed systematically. Special consideration will 
be given to the principal DMI decision makers and others they may 
designate concerning their expectations from the new DMI assessment 
system. 

The background study will be conducted in close coordination with 
the DMI staff and they may be assisted by such consultants as they 
choose. 

Phase Two (one month) 

The main purpose of Phase II is to write a set of specifications 
for the~new DMI planning and assessment system and to have those 
specifications confirmed by the principal DMI decision makers. 

(9) The team of internal and external synthesizers will prepare 
Report No. 4, which is to be a working paper on objectives to be 
achieved by the new DMI system and specifications to be followed In 
preparing alternative designs for such a system. This report is to 
respond to the findings produced by the initial background study 
(Report No. 3 ) . The specifications will include a statement of the 
format in which subsequent designs are to be presented and criteria for 
evaluating the designs. 

(10) The descriptions and specifications contained in Report No. 4 
will be presented to DMI decision makers for their reactions. Refine- 
ment of the description and specifications will continue until DMI 
decision makers confirm clearly that they are satisfactory. 

(14) The team of synthesizers will adjust Report No. 4 appro- 
priately, given reactions by DMI decision makers. 

(11) The coordinator of the study to generate and assess alternative 
systemn for DMI will continue to train members of the advocate and 
convergence teams by providing them with Reports I through 4. 

(12) and (13) Accordingly, the advocate and convergence teams will 
study information made available to them concerning the initial back- 
ground study and DMI decisions based upon the study. 
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Phase Three (one and one-half months) 



The main purpose of Phase III is to design and eva luate two 
alternative planning and assessment systems for DML 

(15) The "advocate team" approach will be applied. Two teams of 
different disciplinary and theoretical composition wi 1 1 each design 

a system to meet the specifications prepared in Phase II. At this 
point Report No. 5, the study design for generating and assessing 
alternative advocate team reports, will be completed. 

(16) This study design will be reviewed by the team of synthesizers 
to insure that the design responds appropriately to the objectives for 
the new DM I system and the specifications for generation of alternative 
assessment systems to meet those objectives, as given in Step 10 in 
Phase 1 1 . 

(17) The coordinator of the study to generate and assess alterna- 
tive DMI systems will complete the training of advocate teams so that 
they have a grasp of the Information contained In Reports 1 through 5« 

(18) The advocate teams will generate Reports Nos. 6 and J, Each 
team will develop its system In relative isolation from the other team. 

(19) The study team for the generation *nd evaluation of alternative 
DMI systems will coordinate a comprehensive evaluation of the alternative 
systems proposed in Reports 6 and 7« The main point of this step is to 
assess the extent to which the competing designs respond appropriately 

to DMI objectives and system specifications, as given in Step 10. In 
performing this evaluation, data will be gathered from DMI staff, 
principal DMI decision makers, representatives of the CEDaR group, 
and others which DMI decision makers may designate. 

(20) Report No. 8, to contain an Initial evaluation of the alter- 
native systems, will be prepared and submitted to the team of 
synthesizers. 

Phase Four (one month) 

The main purpose of this phase is to obtain initia l reactions by 
DMI decision makers to the two advocate team repo rts and to generate yet 
a third strategy which responds to the DM I deci sion maker reactions. 

(21) Based upon Raport No. 8 (initial findings concerning the 
relative strengths and weaknesses of tne two competing DMI systems), 
the synthesis team will prepare Report No. 9» a working paper on the 
selection of one of the alternatives ever the other. 
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(22) This working paper, along with Report No. 8, will be presented 
to the DMI decision makers for their initial reactions and suggestions. 

(23) Based upon DMI decision maker reactions, the training for the 
members of the convergence team will be completed such that they have 

an adequate grasp of all nine reports to date. 

(24) The convergence team will generate Report No. 10 which will 
provide a third strategy for a 0M1 planning and assessment system. It 
is to be recalled that the convergence team will include persons from 
within DMI and within Labs and Centers. This team may confirm the 
selection of one of the advocate reports and elaborate upon it, reject 
both reports and generate a third alternative, or combine what they 
consider to be the best features of both reports. 

(25) Based upon the convergence team report (No. 10), the study 
team to generate and assess alternative systems will perform a final 
assessment of the three competing strategies as reflected in Reports 

6 and 7 and will prepare the final study report. This report (No. 11; 
will be submitted to the synthesis team. 

Phase Five (one month) 

The main purpose of the final phase is to assist PHI decision makers 
to choose a system design and to make deci sions concerning the installa* 
tion of the new system . 

(26) The synthesis team will prepare Report No. 12, which will be 
a proposal for the installation of the new DMI system. This final 
report is to be based upon the preceding eleven reports, and in effect 
is to suggest the decisions that need to be made by DN1, given the 
findings from the previous four stages, in Installing a new system. 

In the performance of this task the synthesis team should feel * re ® to 
call upon the assistance of any of those who have participated in the 
overall study up to this point. 

(2?) The final projected step calls for the DMI decision makers to 
formalize their decisions regarding the installation of the new DMI 
system and to proceed with the implementation of those decisions# 
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Exhibit B 



Ful 1 Term (6jr Months) Budget 



DIRECT COSTS 

1. Project Supervisor 

2. Project Director 

3. Background Evaluator 

4. Alternatives Evaluator 

5. Information Special i st 

6. Technical Writer 

7. Measurement 6- Statistics Specialist 

8. Synthesizer 

9. Advocate Team Members 

10. Convergence Team Members 

1 1 . Secretary 

12. Alternatives Evaluation Respondents 

13. Advisory Panel 
SUB TOTAL 



$1,434 

3,600 

2,000 

6,000 

3,000 

2,000 

2,000 

4.000 

12.000 
6,000 
3,336 
1 ,250 
3,150 
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OTHER COSTS 
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15. Travel 
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SUB TOTAL 



3,692 
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3,690 

300 
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TOTAL 

21. TV-al Subject to Indirect Costs 29,370 
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23. Funds Requested 



22,282 

78,052 

13,722 

91,774 




* 
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Exhibit C 



Budget Notes 



1. Project Supervisor 

(12 month Salary 26,520 ~ 2*210 monthly, 
10% time for 6£ months) 

2. Project Director 

(12 month Salary 17,724 » 1,477 monthly, 
full time for 6£ months) 

3. Background Evaluator 

(12 month Salary 12,000 * 1 ,000 monthly, 
full time for 2 months) 

4. Alternatives Evaluator 

(12 month Salary 12,000 « 1,000 monthly, 
full time for 6 months) 

5. Information Specialist. 

(12 month Salary 12,000 ■* 1 ,000 monthly, 
half time for 6 months) 

6. Technical Writer 

(12 month Salary 12,000 « 1,000 monthly, 
30% time for 6^ months) 

7. Measurement and Statistics Specialist 
(12 month Salary 12,000 « 1,000 monthly, 
30% time for 6£ months) 

8. Synthesizer 
(daily 200, 20 days) 

9. Advocate Team members, 10 
(daily 150, 8 days) 

10. Convergence Team members, 5 
(daily 150, 8 days) 

11. Secretary 

(12 month Salary 6,156 « 513 monthly, 
full time for 65 months) 



ERLC 



9,600 

2,000 

6,000 

3,000 

2,000 

2,000 

4.000 
12,000 

6.000 

3,336 
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Budget Notes - page 2 



12. Alternatives Evaluation Respondents 
(50 @ 25) 

13. Advisory panel for Interim Evaluation 
(daily 1 50, 3 persons, 7 days each) 

SUB TOTAL 

14. Employee Services and Benefits 
(State employees retirement, 

12.9% for 1, 2, 3, 4, 5, 6, 7 
10% for 11) 

15. Travel 

(15 trips by staffs 100, 

15 trips by advocate and convergence teams 
and 6 trips by synthesizer @ 200 
21 trips by advisory panel for interim 
evaluation system 200) 

16. Off Ice suppl ies, reproduction, 
communications, etc. 

17. Per diem 

(123 days 30) 

1 8. Conference f ac i 1 1 1 ies 

19. Subcontract for instrument construction 
and data analysis 

20. Twelve reports <3> 200 

SUB TOTAL - Indirect Costs 

22. Indirect Costs 

(46.72% of 1, 2, 3, 4, 5, 6, 7, 11 
for on-campus personnel) 



$ 1,250 

3,150 

55,770 



3,692 



9,900 

800 

3,690 

300 

1,500 

2,400 

22,282 



‘ 13,722 



23 



Funds Requested 



91,774 



